|
legup-4.0
|
#include "X86ISelLowering.h"#include "Utils/X86ShuffleDecode.h"#include "X86CallingConv.h"#include "X86InstrBuilder.h"#include "X86MachineFunctionInfo.h"#include "X86TargetMachine.h"#include "X86TargetObjectFile.h"#include "llvm/ADT/SmallSet.h"#include "llvm/ADT/Statistic.h"#include "llvm/ADT/StringExtras.h"#include "llvm/ADT/StringSwitch.h"#include "llvm/ADT/VariadicFunction.h"#include "llvm/CodeGen/IntrinsicLowering.h"#include "llvm/CodeGen/MachineFrameInfo.h"#include "llvm/CodeGen/MachineFunction.h"#include "llvm/CodeGen/MachineInstrBuilder.h"#include "llvm/CodeGen/MachineJumpTableInfo.h"#include "llvm/CodeGen/MachineModuleInfo.h"#include "llvm/CodeGen/MachineRegisterInfo.h"#include "llvm/IR/CallSite.h"#include "llvm/IR/CallingConv.h"#include "llvm/IR/Constants.h"#include "llvm/IR/DerivedTypes.h"#include "llvm/IR/Function.h"#include "llvm/IR/GlobalAlias.h"#include "llvm/IR/GlobalVariable.h"#include "llvm/IR/Instructions.h"#include "llvm/IR/Intrinsics.h"#include "llvm/MC/MCAsmInfo.h"#include "llvm/MC/MCContext.h"#include "llvm/MC/MCExpr.h"#include "llvm/MC/MCSymbol.h"#include "llvm/Support/CommandLine.h"#include "llvm/Support/Debug.h"#include "llvm/Support/ErrorHandling.h"#include "llvm/Support/MathExtras.h"#include "llvm/Target/TargetOptions.h"#include <bitset>#include <numeric>#include <cctype>#include "X86GenCallingConv.inc"Go to the source code of this file.
Data Structures | |
| struct | IntrinsicData |
Macros | |
| #define | DEBUG_TYPE "x86-isel" |
Enumerations | |
| enum | StructReturnType { NotStructReturn, RegStructReturn, StackStructReturn } |
| enum | IntrinsicType { GATHER, SCATTER, PREFETCH, RDSEED, RDRAND, RDPMC, RDTSC, XTEST } |
Variables | |
| static cl::opt< bool > | ExperimentalVectorWideningLegalization ("x86-experimental-vector-widening-legalization", cl::init(false), cl::desc("Enable an experimental vector type legalization through widening ""rather than promotion."), cl::Hidden) |
| static cl::opt< bool > | ExperimentalVectorShuffleLowering ("x86-experimental-vector-shuffle-lowering", cl::init(false), cl::desc("Enable an experimental vector shuffle lowering code path."), cl::Hidden) |
| std::map< unsigned, IntrinsicData > | IntrMap |
| #define DEBUG_TYPE "x86-isel" |
Definition at line 57 of file X86ISelLowering.cpp.
| enum IntrinsicType |
| Enumerator | |
|---|---|
| GATHER | |
| SCATTER | |
| PREFETCH | |
| RDSEED | |
| RDRAND | |
| RDPMC | |
| RDTSC | |
| XTEST | |
Definition at line 14385 of file X86ISelLowering.cpp.
| enum StructReturnType |
CallIsStructReturn - Determines whether a call uses struct return semantics.
| Enumerator | |
|---|---|
| NotStructReturn | |
| RegStructReturn | |
| StackStructReturn | |
Definition at line 2122 of file X86ISelLowering.cpp.
|
static |
Tiny helper function to test whether adjacent masks are sequential.
Definition at line 7897 of file X86ISelLowering.cpp.
|
static |
ArgsAreStructReturn - Determines whether a function uses struct return semantics.
Definition at line 2143 of file X86ISelLowering.cpp.
|
static |
Definition at line 9608 of file X86ISelLowering.cpp.
|
static |
Definition at line 5913 of file X86ISelLowering.cpp.
|
static |
Definition at line 2128 of file X86ISelLowering.cpp.
CanFoldXORWithAllOnes - Test whether the XOR operand is a AllOnes vector so it can be folded inside ANDNP.
Definition at line 20479 of file X86ISelLowering.cpp.
|
static |
Try to turn a VSETULT into a VSETULE by modifying its second operand Op1. If non-trivial (for example because it's not constant) return an empty value.
Definition at line 12143 of file X86ISelLowering.cpp.
|
static |
Definition at line 17422 of file X86ISelLowering.cpp.
|
static |
Definition at line 19832 of file X86ISelLowering.cpp.
|
static |
Definition at line 22253 of file X86ISelLowering.cpp.
|
static |
Definition at line 20379 of file X86ISelLowering.cpp.
|
static |
Search for a combinable shuffle across a chain ending in pshufd.
We walk up the chain and look for a combinable shuffle, skipping over shuffles that we could hoist this shuffle's transformation past without altering anything.
Definition at line 18496 of file X86ISelLowering.cpp.
|
static |
Search for a combinable shuffle across a chain ending in pshuflw or pshufhw.
We walk up the chain, skipping shuffles of the other half and looking through shuffles which switch halves trying to find a shuffle of the same pair of dwords.
Definition at line 18609 of file X86ISelLowering.cpp.
|
static |
CommuteVectorShuffleMask - Change values in a shuffle permute mask assuming the two vector operands have swapped position.
Definition at line 3849 of file X86ISelLowering.cpp.
|
static |
Definition at line 4038 of file X86ISelLowering.cpp.
|
static |
Concat two 128-bit vectors into a 256 bit vector using VINSERTF128 instructions. This is used because creating CONCAT_VECTOR nodes of BUILD_VECTORS returns a larger BUILD_VECTOR while we're trying to lower large BUILD_VECTORS.
Definition at line 180 of file X86ISelLowering.cpp.
|
static |
Definition at line 187 of file X86ISelLowering.cpp.
|
static |
CreateCopyOfByValArgument - Make a copy of an aggregate at address specified by "Src" to address "Dst" with size and alignment information specified by the specific parameter attribute. The copy will be passed as a byval function parameter.
Definition at line 2160 of file X86ISelLowering.cpp.
|
static |
Definition at line 194 of file X86ISelLowering.cpp.
|
static |
EltsFromConsecutiveLoads - Given the initializing elements 'Elts' of a vector of type 'VT', see if the elements can be replaced by a single large load which has the same value as a build_vector whose operands are 'elts'.
Example: <load i32 *a, load i32 *a+4, undef, undef> -> zextload a
FIXME: we'd also like to handle the case where the last elements are zero rather than undef via VZEXT_LOAD, but we do not detect that case today. There's even a handy isZeroNode for that purpose.
Definition at line 5643 of file X86ISelLowering.cpp.
|
static |
Definition at line 17059 of file X86ISelLowering.cpp.
|
static |
Definition at line 17024 of file X86ISelLowering.cpp.
|
static |
Definition at line 16987 of file X86ISelLowering.cpp.
|
static |
EmitTailCallStoreRetAddr - Emit a store of the return address if tail call optimization is performed and it is required (FPDiff!=0).
Definition at line 2566 of file X86ISelLowering.cpp.
|
static |
Utility function to emit xbegin specifying the start of an RTM region.
Definition at line 16929 of file X86ISelLowering.cpp.
|
static |
Emit a sequence of two 128-bit horizontal add/sub followed by a concat_vector.
This is a helper function of PerformBUILD_VECTORCombine. This function expects two 256-bit vectors called V0 and V1. At first, each vector is split into two separate 128-bit vectors. Then, the resulting 128-bit vectors are used to implement two horizontal binary operations.
The kind of horizontal binary operation is defined by X86Opcode.
Mode specifies how the 128-bit parts of V0 and V1 are passed in input to the two new horizontal binop. When Mode is set, the first horizontal binop dag node would take as input the lower 128-bit of V0 and the upper 128-bit of V0. The second horizontal binop dag node would take as input the lower 128-bit of V1 and the upper 128-bit of V1. Example: HADD V0_LO, V0_HI HADD V1_LO, V1_HI
Otherwise, the first horizontal binop dag node takes as input the lower 128-bit of V0 and the lower 128-bit of V1, and the second horizontal binop dag node takes the the upper 128-bit of V0 and the upper 128-bit of V1. Example: HADD V0_LO, V1_LO HADD V0_HI, V1_HI
If isUndefLO is set, then the algorithm propagates UNDEF to the lower 128-bits of the result. If isUndefHI is set, then UNDEF is propagated to the upper 128-bits of the result.
Definition at line 6187 of file X86ISelLowering.cpp.
|
static |
Generate a DAG to grab 128-bits from a vector > 128 bits. This sets things up to match to an AVX VEXTRACTF128 / VEXTRACTI128 or AVX-512 VEXTRACTF32x4 / VEXTRACTI32x4 instructions or a simple subregister reference. Idx is an index in the 128 bits we want. It need not be aligned to a 128-bit bounday. That makes lowering EXTRACT_VECTOR_ELT operations easier.
Definition at line 118 of file X86ISelLowering.cpp.
|
static |
Generate a DAG to grab 256-bits from a 512-bit vector.
Definition at line 126 of file X86ISelLowering.cpp.
|
static |
Definition at line 76 of file X86ISelLowering.cpp.
|
static |
FuncIsMadeTailCallSafe - Return true if the function is being made into a tailcall target by changing its ABI.
Definition at line 2197 of file X86ISelLowering.cpp.
|
static |
Definition at line 4696 of file X86ISelLowering.cpp.
|
static |
Definition at line 14204 of file X86ISelLowering.cpp.
|
static |
Definition at line 9028 of file X86ISelLowering.cpp.
|
static |
Definition at line 4711 of file X86ISelLowering.cpp.
|
static |
getLegalSplat - Generate a legal splat with supported x86 shuffles
Definition at line 5022 of file X86ISelLowering.cpp.
|
static |
getMaxByValAlign - Helper for getByValTypeAlignment to determine the desired ByVal argument alignment.
Definition at line 1646 of file X86ISelLowering.cpp.
|
static |
Definition at line 8907 of file X86ISelLowering.cpp.
|
static |
Definition at line 8937 of file X86ISelLowering.cpp.
getMOVLMask - Returns a vector_shuffle mask for an movs{s|d}, movd operation of specified width.
Definition at line 4966 of file X86ISelLowering.cpp.
|
static |
Definition at line 8918 of file X86ISelLowering.cpp.
|
static |
Definition at line 8953 of file X86ISelLowering.cpp.
|
static |
getNumOfConsecutiveZeros - Return the number of elements of a vector shuffle operation which come from a consecutively from a zero. The search can start in two different directions, from left or right. We count undefs as zeros until PreferredNum is reached.
Definition at line 5254 of file X86ISelLowering.cpp.
|
static |
getOnesVector - Returns a vector of specified type with all bits set. Always build ones vectors as <4 x i32> or <8 x i32>. For 256-bit types with no AVX2 supprt, use two <4 x i32> inserted in a <8 x i32> appropriately. Then bitcast to their original type, ensuring they get CSE'd.
Definition at line 4932 of file X86ISelLowering.cpp.
|
static |
Definition at line 14254 of file X86ISelLowering.cpp.
|
static |
Generate a PSHUFB if possible. Selects elements from V1 according to MaskVals. MaskVals[OutputIdx] = InputIdx specifies that we want to shuffle the element at InputIdx in V1 to OutputIdx in the result. If MaskVals refers to elements outside of V1 or is undef (-1), insert a zero.
Definition at line 8097 of file X86ISelLowering.cpp.
|
static |
Get the PSHUF-style mask from PSHUF node.
This is a very minor wrapper around getTargetShuffleMask to easy forming v4 PSHUF-style masks that can be reused with such instructions.
Definition at line 18468 of file X86ISelLowering.cpp.
|
static |
Definition at line 14279 of file X86ISelLowering.cpp.
|
static |
Definition at line 14325 of file X86ISelLowering.cpp.
|
static |
Definition at line 14231 of file X86ISelLowering.cpp.
|
static |
getShufflePALIGNRImmediate - Return the appropriate immediate to shuffle the specified VECTOR_SHUFFLE mask with the PALIGNR instruction.
Definition at line 4673 of file X86ISelLowering.cpp.
|
static |
getShufflePSHUFHWImmediate - Return the appropriate immediate to shuffle the specified VECTOR_SHUFFLE mask with the PSHUFHW instruction.
Definition at line 4625 of file X86ISelLowering.cpp.
|
static |
getShufflePSHUFLWImmediate - Return the appropriate immediate to shuffle the specified VECTOR_SHUFFLE mask with the PSHUFLW instruction.
Definition at line 4649 of file X86ISelLowering.cpp.
|
static |
getShuffleScalarElt - Returns the scalar element that will make up the ith element of the result of the vector shuffle.
Definition at line 5188 of file X86ISelLowering.cpp.
|
static |
getShuffleSHUFImmediate - Return the appropriate immediate to shuffle the specified VECTOR_SHUFFLE mask with PSHUF* and SHUFP* instructions. Handles 128-bit and 256-bit.
Definition at line 4595 of file X86ISelLowering.cpp.
|
static |
getShuffleVectorZeroOrUndef - Return a vector_shuffle of the specified vector of zero or undef vector. This produces a shuffle where the low element of V2 is swizzled into the zero/undef vector, landing at element Idx. This produces a shuffle mask like 4,1,2,3 (idx=0) or 0,1,2,4 (idx=3).
Definition at line 5090 of file X86ISelLowering.cpp.
|
static |
getShuffleVPERM2X128Immediate - Return the appropriate immediate to shuffle the specified VECTOR_MASK mask with VPERM2F128/VPERM2I128 instructions.
Definition at line 4334 of file X86ISelLowering.cpp.
|
static |
getTargetShuffleMask - Calculates the shuffle mask corresponding to the target specific opcode. Returns true if the Mask could be calculated. Sets IsUnary to true if only uses one source.
Definition at line 5108 of file X86ISelLowering.cpp.
|
static |
Definition at line 3405 of file X86ISelLowering.cpp.
|
static |
Definition at line 3416 of file X86ISelLowering.cpp.
|
static |
Definition at line 3430 of file X86ISelLowering.cpp.
|
static |
Definition at line 3443 of file X86ISelLowering.cpp.
|
static |
Definition at line 13375 of file X86ISelLowering.cpp.
|
static |
Definition at line 13451 of file X86ISelLowering.cpp.
|
static |
Definition at line 10339 of file X86ISelLowering.cpp.
For an EXTRACT_VECTOR_ELT with a constant index return the real underlying vector and index.
Modifies ExtractedFromVec to the real vector and returns the real index.
Definition at line 5883 of file X86ISelLowering.cpp.
getUnpackh - Returns a vector_shuffle node for an unpackh operation.
Definition at line 4989 of file X86ISelLowering.cpp.
getUnpackl - Returns a vector_shuffle node for an unpackl operation.
Definition at line 4977 of file X86ISelLowering.cpp.
|
static |
Get a 4-lane 8-bit shuffle immediate for a mask.
This helper function produces an 8-bit shuffle immediate corresponding to the ubiquitous shuffle encoding scheme used in x86 instructions for shuffling 4 lanes. It can be used with most of the PSHUF instructions for example.
NB: We rely heavily on "undef" masks preserving the input lane.
Definition at line 6928 of file X86ISelLowering.cpp.
|
static |
getVShift - Return a vector logical shift node.
Definition at line 5547 of file X86ISelLowering.cpp.
|
static |
getVZextMovL - Return a zero-extending vector move low node.
Definition at line 8604 of file X86ISelLowering.cpp.
|
static |
getZeroVector - Returns a vector of specified type with all zero elements.
Definition at line 4885 of file X86ISelLowering.cpp.
|
static |
hasFPCMov - is there a floating point cmov for the specific X86 condition code. Current x86 isa includes the following FP cmov instructions: fcmovb, fcomvbe, fcomve, fcmovu, fcmovae, fcmova, fcmovne, fcmovnu.
Definition at line 3643 of file X86ISelLowering.cpp.
|
static |
return true if Op has a use that doesn't just read flags.
Definition at line 11634 of file X86ISelLowering.cpp.
|
static |
Definition at line 14398 of file X86ISelLowering.cpp.
|
static |
Generate a DAG to put 128-bits into a vector > 128 bits. This sets things up to match to an AVX VINSERTF128/VINSERTI128 or AVX-512 VINSERTF32x4/VINSERTI32x4 instructions or a simple superregister reference. Idx is an index in the 128 bits we want. It need not be aligned to a 128-bit bounday. That makes lowering INSERT_VECTOR_ELT operations easier.
Definition at line 162 of file X86ISelLowering.cpp.
|
static |
Definition at line 169 of file X86ISelLowering.cpp.
|
static |
Definition at line 132 of file X86ISelLowering.cpp.
|
static |
Definition at line 11949 of file X86ISelLowering.cpp.
|
static |
Definition at line 12817 of file X86ISelLowering.cpp.
|
static |
Definition at line 8009 of file X86ISelLowering.cpp.
|
static |
Return true if the calling convention is a C calling convention.
Definition at line 2178 of file X86ISelLowering.cpp.
|
static |
isCommutedMOVLMask - Returns true if the shuffle mask is except the reverse of what x86 movss want. X86 movs requires the lowest element to be lowest element of vector 2 and the other elements to come from vector 1 in order.
Definition at line 4433 of file X86ISelLowering.cpp.
|
static |
Return true if N implements a horizontal binop and return the operands for the horizontal binop into V0 and V1.
This is a helper function of PerformBUILD_VECTORCombine. This function checks that the build_vector N in input implements a horizontal operation. Parameter Opcode defines the kind of horizontal operation to match. For example, if Opcode is equal to ISD::ADD, then this function checks if N implements a horizontal arithmetic add; if instead Opcode is equal to ISD::SUB, then this function checks if this is a horizontal arithmetic sub.
This function only analyzes elements of N whose indices are in range [BaseIdx, LastIdx).
Definition at line 6079 of file X86ISelLowering.cpp.
isHorizontalBinOp - Return 'true' if this vector operation is "horizontal" and return the operands for the horizontal operation in LHS and RHS. A horizontal operation performs the binary operation on successive elements of its first operand, then on successive elements of its second operand, returning the resulting values in a vector. For example, if A = < float a0, float a1, float a2, float a3 > and B = < float b0, float b1, float b2, float b3 > then the result of doing a horizontal operation on A and B is A horizontal-op B = < a0 op a1, a2 op a3, b0 op b1, b2 op b3 >. In short, LHS and RHS are inspected to see if LHS op RHS is of the form A horizontal-op B, for some already available A and B, and if so then LHS is set to A, RHS to B, and the routine returns 'true'. Note that the binary operation should have the property that if one of the operands is UNDEF then the result is UNDEF.
Definition at line 21305 of file X86ISelLowering.cpp.
Definition at line 4255 of file X86ISelLowering.cpp.
isINSERTPSMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to INSERTPS. i. e: If all but one element come from the same vector.
Definition at line 4006 of file X86ISelLowering.cpp.
isMOVDDUPMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to 128-bit version of MOVDDUP.
Definition at line 4525 of file X86ISelLowering.cpp.
isMOVDDUPYMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to 256-bit version of MOVDDUP.
Definition at line 4505 of file X86ISelLowering.cpp.
isMOVHLPS_v_undef_Mask - Special case of isMOVHLPSMask for canonical form of vector_shuffle v, v, <2, 3, 2, 3>, i.e. vector_shuffle v, undef, <2, 3, 2, 3>
Definition at line 3944 of file X86ISelLowering.cpp.
isMOVHLPSMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to MOVHLPS.
Definition at line 3925 of file X86ISelLowering.cpp.
isMOVLHPSMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to MOVLHPS.
Definition at line 3983 of file X86ISelLowering.cpp.
isMOVLMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to MOVSS, MOVSD, and MOVD, i.e. setting the lowest element.
Definition at line 4279 of file X86ISelLowering.cpp.
isMOVLPMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to MOVLP{S|D}.
Definition at line 3961 of file X86ISelLowering.cpp.
|
static |
isMOVSHDUPMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to MOVSHDUP. Masks to match: <1, 1, 3, 3> or <1, 1, 3, 3, 5, 5, 7, 7>
Definition at line 4457 of file X86ISelLowering.cpp.
|
static |
isMOVSLDUPMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to MOVSLDUP. Masks to match: <0, 0, 2, 2> or <0, 0, 2, 2, 4, 4, 6, 6>
Definition at line 4481 of file X86ISelLowering.cpp.
|
static |
Tiny helper function to identify a no-op mask.
This is a somewhat boring predicate function. It checks whether the mask array input, which is assumed to be a single-input shuffle mask of the kind used by the X86 shuffle instructions (not a fully general ShuffleVectorSDNode mask) requires any shuffles to occur. Both undef and an in-place shuffle are 'no-op's.
Definition at line 6898 of file X86ISelLowering.cpp.
|
static |
isPALIGNRMask - Return true if the node specifies a shuffle of elements that is suitable for input to PALIGNR.
Definition at line 3776 of file X86ISelLowering.cpp.
Definition at line 4357 of file X86ISelLowering.cpp.
isPSHUFDMask - Return true if the node specifies a shuffle of elements that is suitable for input to PSHUFD or PSHUFW. That is, it doesn't reference the second operand.
Definition at line 3708 of file X86ISelLowering.cpp.
isPSHUFHWMask - Return true if the node specifies a shuffle of elements that is suitable for input to PSHUFHW.
Definition at line 3718 of file X86ISelLowering.cpp.
isPSHUFLWMask - Return true if the node specifies a shuffle of elements that is suitable for input to PSHUFLW.
Definition at line 3747 of file X86ISelLowering.cpp.
|
static |
isScalarLoadToVector - Returns true if the node is a scalar load that is promoted to a vector. It also returns the LoadSDNode by reference if required.
Definition at line 4791 of file X86ISelLowering.cpp.
|
static |
isSequentialOrUndefInRange - Return true if every element in Mask, beginning from position Pos and ending in Pos+Size, falls within the specified sequential range (L, L+Pos]. or is undef.
Definition at line 3697 of file X86ISelLowering.cpp.
|
static |
isShuffleHigh128VectorInsertLow - Checks whether the shuffle node is the same as extracting the high 128-bit part of 256-bit vector and then inserting the result into the low part of a new 256-bit vector
Definition at line 18340 of file X86ISelLowering.cpp.
|
static |
isShuffleLow128VectorInsertHigh - Checks whether the shuffle node is the same as extracting the low 128-bit part of 256-bit vector and then inserting the result into the high part of a new 256-bit vector
Definition at line 18356 of file X86ISelLowering.cpp.
|
static |
isShuffleMaskConsecutive - Check if the shuffle mask indicies [MaskI, MaskE) correspond consecutively to elements from one of the vector operands, starting from its index OpIdx. Also tell OpNum which source vector operand.
Definition at line 5280 of file X86ISelLowering.cpp.
isSHUFPMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to 128/256-bit SHUFPS and SHUFPD. If Commuted is true, then it checks for sources to be reverse of what x86 shuffles want.
Definition at line 3866 of file X86ISelLowering.cpp.
|
static |
Helper function to classify a mask as a single-input mask.
This isn't a generic single-input test because in the vector shuffle lowering we canonicalize single inputs to be the first input operand. This means we can more quickly test for a single input by only checking whether an input from the second operand exists. We also assume that the size of mask corresponds to the size of the input vectors which isn't true in the fully general case.
Definition at line 6913 of file X86ISelLowering.cpp.
|
static |
IsTailCallConvention - Return true if the calling convention is one that supports tail call optimization.
Definition at line 2172 of file X86ISelLowering.cpp.
|
static |
Definition at line 3378 of file X86ISelLowering.cpp.
|
static |
Definition at line 12506 of file X86ISelLowering.cpp.
|
static |
isUndefOrEqual - Val is either less than zero (undef) or equal to the specified value.
Definition at line 3690 of file X86ISelLowering.cpp.
|
static |
isUndefOrInRange - Return true if Val is undef or if its value falls within the specified range (L, H].
Definition at line 3684 of file X86ISelLowering.cpp.
isUNPCKH_v_undef_Mask - Special case of isUNPCKHMask for canonical form of vector_shuffle v, v, <2, 6, 3, 7>, i.e. vector_shuffle v, undef, <2, 2, 3, 3>
Definition at line 4222 of file X86ISelLowering.cpp.
|
static |
isUNPCKHMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to UNPCKH.
Definition at line 4131 of file X86ISelLowering.cpp.
isUNPCKL_v_undef_Mask - Special case of isUNPCKLMask for canonical form of vector_shuffle v, v, <0, 4, 1, 5>, i.e. vector_shuffle v, undef, <0, 0, 1, 1>
Definition at line 4179 of file X86ISelLowering.cpp.
|
static |
isUNPCKLMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to UNPCKL.
Definition at line 4083 of file X86ISelLowering.cpp.
|
static |
isVectorShift - Returns true if the shuffle can be implemented as a logical left or right shift of a vector.
Definition at line 5378 of file X86ISelLowering.cpp.
|
static |
isVectorShiftLeft - Returns true if the shuffle can be implemented as a logical left shift of a vector.
Definition at line 5343 of file X86ISelLowering.cpp.
|
static |
isVectorShiftRight - Returns true if the shuffle can be implemented as a logical left shift of a vector.
Definition at line 5308 of file X86ISelLowering.cpp.
|
static |
isVEXTRACTIndex - Return true if the specified EXTRACT_SUBVECTOR operand specifies a vector extract that is suitable for instruction that extract 128 or 256 bit vectors
Definition at line 4542 of file X86ISelLowering.cpp.
|
static |
isVINSERTIndex - Return true if the specified INSERT_SUBVECTOR operand specifies a subvector insert that is suitable for input to insertion of 128 or 256-bit subvectors
Definition at line 4561 of file X86ISelLowering.cpp.
isVPERM2X128Mask - Match 256-bit shuffles where the elements are considered as permutations between 128-bit chunks or halves. As an example: this shuffle bellow: vector_shuffle <4, 5, 6, 7, 12, 13, 14, 15> The first half comes from the second half of V1 and the second half from the the second half of V2.
Definition at line 4303 of file X86ISelLowering.cpp.
isVPERMILPMask - Return true if the specified VECTOR_SHUFFLE operand specifies a shuffle of elements that is suitable for input to VPERMILPD*. Note that VPERMIL mask matching is different depending whether theunderlying type is 32 or 64. In the VPERMILPS the high half of the mask should point to the same elements of the low, but to the higher half of the source. In VPERMILPD the two lanes could be shuffled independently of each other with the same restriction that lanes can't be crossed. Also handles PSHUFDY.
Definition at line 4401 of file X86ISelLowering.cpp.
|
static |
Return true if the condition is an unsigned comparison operation.
Definition at line 3535 of file X86ISelLowering.cpp.
|
static |
Definition at line 12481 of file X86ISelLowering.cpp.
|
static |
Definition at line 12829 of file X86ISelLowering.cpp.
|
static |
isZero - Returns true if Elt is a constant integer zero
Definition at line 4755 of file X86ISelLowering.cpp.
|
static |
isZeroShuffle - Returns true if N is a VECTOR_SHUFFLE that can be resolved to an zero vector. FIXME: move to dag combiner / method on ShuffleVectorSDNode
Definition at line 4858 of file X86ISelLowering.cpp.
|
static |
Dispatching routine to lower various 128-bit x86 vector shuffles.
This routine breaks down the specific type of 128-bit shuffle and dispatches to the lowering routines accordingly.
Definition at line 7874 of file X86ISelLowering.cpp.
|
static |
Definition at line 14993 of file X86ISelLowering.cpp.
|
static |
Definition at line 12072 of file X86ISelLowering.cpp.
|
static |
Definition at line 15020 of file X86ISelLowering.cpp.
|
static |
Definition at line 16128 of file X86ISelLowering.cpp.
|
static |
Definition at line 14690 of file X86ISelLowering.cpp.
|
static |
Definition at line 11154 of file X86ISelLowering.cpp.
|
static |
Definition at line 5561 of file X86ISelLowering.cpp.
|
static |
Definition at line 15959 of file X86ISelLowering.cpp.
|
static |
Definition at line 16103 of file X86ISelLowering.cpp.
|
static |
Definition at line 6842 of file X86ISelLowering.cpp.
|
static |
Definition at line 11080 of file X86ISelLowering.cpp.
|
static |
Definition at line 16036 of file X86ISelLowering.cpp.
|
static |
LowerBuildVectorv16i8 - Custom lower build_vector of v16i8.
Definition at line 5394 of file X86ISelLowering.cpp.
|
static |
LowerBuildVectorv4x32 - Custom lower build_vector of v4i32 or v4f32.
Definition at line 5473 of file X86ISelLowering.cpp.
|
static |
LowerBuildVectorv8i16 - Custom lower build_vector of v8i16.
Definition at line 5442 of file X86ISelLowering.cpp.
|
static |
Definition at line 15995 of file X86ISelLowering.cpp.
|
static |
Definition at line 6866 of file X86ISelLowering.cpp.
|
static |
Definition at line 14912 of file X86ISelLowering.cpp.
|
static |
Definition at line 14946 of file X86ISelLowering.cpp.
|
static |
Definition at line 14971 of file X86ISelLowering.cpp.
|
static |
Definition at line 10087 of file X86ISelLowering.cpp.
|
static |
Definition at line 9708 of file X86ISelLowering.cpp.
|
static |
Definition at line 11378 of file X86ISelLowering.cpp.
|
static |
Definition at line 11449 of file X86ISelLowering.cpp.
|
static |
Definition at line 11529 of file X86ISelLowering.cpp.
|
static |
Definition at line 11413 of file X86ISelLowering.cpp.
|
static |
Definition at line 11365 of file X86ISelLowering.cpp.
|
static |
Definition at line 16154 of file X86ISelLowering.cpp.
|
static |
Definition at line 10113 of file X86ISelLowering.cpp.
|
static |
Definition at line 9917 of file X86ISelLowering.cpp.
|
static |
Definition at line 14485 of file X86ISelLowering.cpp.
|
static |
Definition at line 13486 of file X86ISelLowering.cpp.
|
static |
Definition at line 12100 of file X86ISelLowering.cpp.
|
static |
Definition at line 16088 of file X86ISelLowering.cpp.
|
static |
Definition at line 15034 of file X86ISelLowering.cpp.
|
static |
Definition at line 15158 of file X86ISelLowering.cpp.
|
static |
Definition at line 14376 of file X86ISelLowering.cpp.
|
static |
Definition at line 10056 of file X86ISelLowering.cpp.
|
static |
Definition at line 15246 of file X86ISelLowering.cpp.
|
static |
Definition at line 15417 of file X86ISelLowering.cpp.
|
static |
Definition at line 15561 of file X86ISelLowering.cpp.
|
static |
LowerShiftParts - Lower SRA_PARTS and friends, which return two i32 values and take a 2 x i32 value to shift plus a shift amount.
Definition at line 10625 of file X86ISelLowering.cpp.
|
static |
Definition at line 12763 of file X86ISelLowering.cpp.
|
static |
Definition at line 12732 of file X86ISelLowering.cpp.
|
static |
Definition at line 15027 of file X86ISelLowering.cpp.
|
static |
Definition at line 10430 of file X86ISelLowering.cpp.
|
static |
Definition at line 10370 of file X86ISelLowering.cpp.
|
static |
Definition at line 10384 of file X86ISelLowering.cpp.
|
static |
Definition at line 10390 of file X86ISelLowering.cpp.
|
static |
Generic lowering of v16i8 shuffles.
This is a hybrid strategy to lower v16i8 vectors. It first attempts to detect any complexity reducing interleaving. If that doesn't help, it uses UNPCK to spread the i8 elements across two i16-element vectors, and uses the existing lowering for v8i16 blends on each half, finally PACK-ing them back together.
Definition at line 7668 of file X86ISelLowering.cpp.
|
static |
Handle lowering of 2-lane 64-bit floating point shuffles.
This is the basis function for the 2-lane 64-bit shuffles as we have full support for floating point shuffles but not integer shuffles. These instructions will incur a domain crossing penalty on some chips though so it is better to avoid lowering through this for integer vectors where possible.
Definition at line 6951 of file X86ISelLowering.cpp.
|
static |
Handle lowering of 2-lane 64-bit integer shuffles.
Tries to lower a 2-lane 64-bit shuffle using shuffle operations provided by the integer unit to minimize domain crossing penalties. However, for blends it falls back to the floating point shuffle operation with appropriate bit casting.
Definition at line 6983 of file X86ISelLowering.cpp.
|
static |
Lower 4-lane 32-bit floating point shuffles.
Uses instructions exclusively from the floating point unit to minimize domain crossing penalties, as these are sufficient to implement all v4f32 shuffles.
Definition at line 7023 of file X86ISelLowering.cpp.
|
static |
Lower 4-lane i32 vector shuffles.
We try to handle these with integer-domain shuffles where we can, but for blends we use the floating point domain blend instructions.
Definition at line 7117 of file X86ISelLowering.cpp.
|
static |
Blend two v8i16 vectors using a naive unpack strategy.
This strategy only works when the inputs from each vector fit into a single half of that vector, and generally there are not so many inputs as to leave the in-place shuffles required highly constrained (and thus expensive). It shifts all the inputs into a single side of both input vectors and then uses an unpack to interleave these inputs in a single vector. At that point, we will fall back on the generic single input shuffle lowering.
Definition at line 7474 of file X86ISelLowering.cpp.
|
static |
Lowering of single-input v8i16 shuffles is the cornerstone of SSE2 shuffle lowering, and the most complex part.
The lowering strategy is to try to form pairs of input lanes which are targeted at the same half of the final vector, and then use a dword shuffle to place them onto the right half, and finally unpack the paired lanes into their final position.
The exact breakdown of how to form these dword pairs and align them on the correct sides is really tricky. See the comments within the function for more of the details.
Definition at line 7157 of file X86ISelLowering.cpp.
|
static |
Generic lowering of 8-lane i16 shuffles.
This handles both single-input shuffles and combined shuffle/blends with two inputs. The single input shuffles are immediately delegated to a dedicated lowering routine.
The blends are lowered in one of three fundamental ways. If there are few enough inputs, it delegates to a basic UNPCK-based strategy. If the shuffle of the input is significantly cheaper when lowered as an interleaving of the two inputs, try to interleave them. Otherwise, blend the low and high halves of the inputs separately (making them have relatively few inputs) and then concatenate them.
Definition at line 7591 of file X86ISelLowering.cpp.
|
static |
Definition at line 13356 of file X86ISelLowering.cpp.
|
static |
LowerVECTOR_SHUFFLE_128v4 - Handle all 128-bit wide vectors with 4 elements, and match them with several different shuffle types.
Definition at line 8753 of file X86ISelLowering.cpp.
|
static |
LowerVECTOR_SHUFFLE_256 - Handle all 256-bit wide vectors shuffles which could not be matched by any known target speficic shuffle
Definition at line 8640 of file X86ISelLowering.cpp.
|
static |
Definition at line 8054 of file X86ISelLowering.cpp.
|
static |
v16i16 shuffles
FIXME: We only support generation of a single pshufb currently. We can generalize the other applicable cases from LowerVECTOR_SHUFFLEv8i16 as well (e.g 2 x pshufb + 1 x por).
Definition at line 8382 of file X86ISelLowering.cpp.
|
static |
Definition at line 8399 of file X86ISelLowering.cpp.
|
static |
Definition at line 8528 of file X86ISelLowering.cpp.
|
static |
Definition at line 8142 of file X86ISelLowering.cpp.
|
static |
Definition at line 11542 of file X86ISelLowering.cpp.
|
static |
LowerVectorBroadcast - Attempt to use the vbroadcast instruction to generate a splat value for the following cases:
Definition at line 5743 of file X86ISelLowering.cpp.
|
static |
Definition at line 9114 of file X86ISelLowering.cpp.
|
static |
Top-level lowering for x86 vector shuffles.
This handles decomposition, canonicalization, and lowering of all x86 vector shuffles. Most of the specific lowering strategies are encapsulated above in helper routines. The canonicalization attempts to widen shuffles to involve fewer lanes of wider elements, consolidate symmetric patterns s.t. only one of the two inputs needs to be tested, etc.
Definition at line 7912 of file X86ISelLowering.cpp.
|
static |
Definition at line 9643 of file X86ISelLowering.cpp.
|
static |
Definition at line 12170 of file X86ISelLowering.cpp.
|
static |
Definition at line 15808 of file X86ISelLowering.cpp.
|
static |
Definition at line 11165 of file X86ISelLowering.cpp.
|
static |
Definition at line 11123 of file X86ISelLowering.cpp.
|
static |
Try to fold a build_vector that performs an 'addsub' into the sequence of 'vadd + vsub + blendi'.
Definition at line 6227 of file X86ISelLowering.cpp.
|
static |
MatchingStackOffset - Return true if the given stack call argument is already available in the same position (relatively) of the caller's incoming argument stack.
Definition at line 3104 of file X86ISelLowering.cpp.
|
static |
Matches a VSELECT onto min/max or return 0 if the node doesn't match.
Definition at line 19132 of file X86ISelLowering.cpp.
|
static |
Definition at line 21768 of file X86ISelLowering.cpp.
|
static |
Definition at line 3374 of file X86ISelLowering.cpp.
|
static |
Definition at line 3370 of file X86ISelLowering.cpp.
|
static |
Definition at line 8892 of file X86ISelLowering.cpp.
|
static |
Definition at line 9009 of file X86ISelLowering.cpp.
|
static |
NormalizeMask - V2 is a splat, modify the mask (if needed) so all elements that point to V2 points to its first element.
Definition at line 4956 of file X86ISelLowering.cpp.
|
static |
Definition at line 9201 of file X86ISelLowering.cpp.
|
static |
Definition at line 21960 of file X86ISelLowering.cpp.
|
static |
Definition at line 21932 of file X86ISelLowering.cpp.
|
static |
PerformADDCombine - Do target-specific dag combines on integer adds.
Definition at line 21997 of file X86ISelLowering.cpp.
|
static |
Definition at line 20590 of file X86ISelLowering.cpp.
|
static |
Definition at line 21826 of file X86ISelLowering.cpp.
|
static |
Definition at line 21509 of file X86ISelLowering.cpp.
|
static |
Definition at line 6346 of file X86ISelLowering.cpp.
|
static |
Optimize X86ISD::CMOV [LHS, RHS, CONDCODE (e.g. X86::COND_NE), CONDVAL].
Definition at line 19950 of file X86ISelLowering.cpp.
|
static |
PerformEXTRACT_VECTOR_ELTCombine - Detect vector gather/scatter index generation and convert it from being a bunch of shuffles and extracts to a simple store and scalar loads to extract the elements.
Definition at line 19036 of file X86ISelLowering.cpp.
|
static |
PerformFADDCombine - Do target-specific dag combines on floating point adds.
Definition at line 21417 of file X86ISelLowering.cpp.
|
static |
PerformFANDCombine - Do target-specific dag combines on X86ISD::FAND nodes.
Definition at line 21484 of file X86ISelLowering.cpp.
|
static |
PerformFANDNCombine - Do target-specific dag combines on X86ISD::FANDN nodes.
Definition at line 21497 of file X86ISelLowering.cpp.
|
static |
Definition at line 21595 of file X86ISelLowering.cpp.
|
static |
PerformFMinFMaxCombine - Do target-specific dag combines on X86ISD::FMIN and X86ISD::FMAX nodes.
Definition at line 21463 of file X86ISelLowering.cpp.
|
static |
PerformFORCombine - Do target-specific dag combines on X86ISD::FOR and X86ISD::FXOR nodes.
Definition at line 21448 of file X86ISelLowering.cpp.
|
static |
PerformFSUBCombine - Do target-specific dag combines on floating point subs.
Definition at line 21432 of file X86ISelLowering.cpp.
|
static |
Definition at line 21738 of file X86ISelLowering.cpp.
|
static |
Definition at line 20816 of file X86ISelLowering.cpp.
|
static |
Definition at line 20121 of file X86ISelLowering.cpp.
|
static |
Definition at line 21683 of file X86ISelLowering.cpp.
|
static |
PerformLOADCombine - Do target-specific dag combines on LOAD nodes.
Definition at line 20866 of file X86ISelLowering.cpp.
|
static |
PerformMulCombine - Optimize a single multiply with constant into two in order to implement it with two cheaper instructions, e.g. LEA + SHL, LEA + LEA.
Definition at line 20226 of file X86ISelLowering.cpp.
|
static |
Definition at line 20654 of file X86ISelLowering.cpp.
|
static |
PerformSELECTCombine - Do target-specific dag combines on SELECT and VSELECT nodes.
Definition at line 19252 of file X86ISelLowering.cpp.
|
static |
Definition at line 21782 of file X86ISelLowering.cpp.
|
static |
Definition at line 21576 of file X86ISelLowering.cpp.
|
static |
PerformShiftCombine - Combine shifts.
Definition at line 20359 of file X86ISelLowering.cpp.
|
static |
Returns a vector of 0s if the node in input is a vector logical shift by a constant amount which is known to be bigger than or equal to the vector element size in bits.
Definition at line 20331 of file X86ISelLowering.cpp.
|
static |
Definition at line 20286 of file X86ISelLowering.cpp.
|
static |
PerformShuffleCombine - Performs several different shuffle combines.
Definition at line 18792 of file X86ISelLowering.cpp.
|
static |
PerformShuffleCombine256 - Performs shuffle combines for 256-bit vectors.
Definition at line 18370 of file X86ISelLowering.cpp.
|
static |
Definition at line 21541 of file X86ISelLowering.cpp.
|
static |
Definition at line 21893 of file X86ISelLowering.cpp.
|
static |
PerformSTORECombine - Do target-specific dag combines on STORE nodes.
Definition at line 21061 of file X86ISelLowering.cpp.
|
static |
Definition at line 22012 of file X86ISelLowering.cpp.
|
static |
Try to combine x86 target specific shuffles.
Definition at line 18687 of file X86ISelLowering.cpp.
|
static |
PerformTruncateCombine - Converts truncate operation to a sequence of vector shuffle operations. It is possible when we truncate 256-bit vector to 128-bit vector
Definition at line 18932 of file X86ISelLowering.cpp.
|
static |
Definition at line 21847 of file X86ISelLowering.cpp.
|
static |
Definition at line 21528 of file X86ISelLowering.cpp.
|
static |
performVZEXTCombine - Performs build vector combines
Definition at line 22046 of file X86ISelLowering.cpp.
|
static |
Definition at line 20850 of file X86ISelLowering.cpp.
|
static |
Definition at line 21635 of file X86ISelLowering.cpp.
|
static |
PromoteSplat - Splat is promoted to target supported vector shuffles.
Definition at line 5048 of file X86ISelLowering.cpp.
|
static |
Definition at line 5004 of file X86ISelLowering.cpp.
|
static |
Definition at line 16295 of file X86ISelLowering.cpp.
|
static |
RewriteAsNarrowerShuffle - Try rewriting v8i16 and v16i8 shuffles as 4 wide ones, or rewriting v4i32 / v4f32 as 2 wide ones if possible. This can be done when every pair / quad of shuffle mask elements point to elements in the right sequence. e.g. vector_shuffle X, Y, <2, 3, | 10, 11, | 0, 1, | 14, 15>
Definition at line 8562 of file X86ISelLowering.cpp.
|
static |
Detect whether the mask pattern should be lowered through interleaving.
This essentially tests whether viewing the mask as an interleaving of two sub-sequences reduces the cross-input traffic of a blend operation. If so, lowering it through interleaving is a significantly better strategy.
Definition at line 7434 of file X86ISelLowering.cpp.
ShouldXformToMOVHLPS - Return true if the node should be transformed to match movhlps. The lower half elements should come from upper half of V1 (and in order), and the upper half elements should come from the upper half of V2 (and in order).
Definition at line 4774 of file X86ISelLowering.cpp.
ShouldXformToMOVLP{S|D} - Return true if the node should be transformed to match movlp{s|d}. The lower half elements should come from lower half of V1 (and in order), and the upper half elements should come from the upper half of V2 (and in order). And since V1 will become the source of the MOVLP, it must be either a vector load or a scalar load to vector.
Definition at line 4830 of file X86ISelLowering.cpp.
|
static |
In vector type VT, return true if the element at index InputIdx falls on a different 128-bit lane than OutputIdx.
Definition at line 8086 of file X86ISelLowering.cpp.
| STATISTIC | ( | NumTailCalls | , |
| "Number of tail calls" | |||
| ) |
|
static |
Definition at line 19206 of file X86ISelLowering.cpp.
|
static |
TranslateX86CC - do a one to one translation of a ISD::CondCode to the X86 specific condition code, returning the condition code and the LHS/RHS of the comparison to make.
Definition at line 3555 of file X86ISelLowering.cpp.
|
static |
Definition at line 12027 of file X86ISelLowering.cpp.
|
static |
Definition at line 20511 of file X86ISelLowering.cpp.
|
static |
Definition at line 4804 of file X86ISelLowering.cpp.
|
static |
XFormVExtractWithShuffleIntoLoad - Check if a vector extract from a target specific shuffle of a load can be folded into a single element load. Similar handling for VECTOR_SHUFFLE is performed by DAGCombiner, but shuffles have been customed lowered so we need to handle those here.
Definition at line 18942 of file X86ISelLowering.cpp.
|
static |
|
static |
| std::map< unsigned, IntrinsicData> IntrMap |
Definition at line 14397 of file X86ISelLowering.cpp.